Skip to content

feat: add pilot database and router#570

Open
Robin-Van-de-Merghel wants to merge 11 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management
Open

feat: add pilot database and router#570
Robin-Van-de-Merghel wants to merge 11 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management

Conversation

@Robin-Van-de-Merghel

Copy link
Copy Markdown
Contributor

Split of #421 , first part : pilot management

@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from 8c655b0 to 2cfe44a Compare June 13, 2025 11:19
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel marked this pull request as draft June 13, 2025 11:36
Comment thread diracx-db/src/diracx/db/sql/utils/functions.py Outdated
Comment thread diracx-logic/src/diracx/logic/auth/token.py
Comment thread diracx-core/src/diracx/core/settings.py
Comment thread diracx-routers/src/diracx/routers/auth/token.py
Comment thread diracx-routers/src/diracx/routers/utils/users.py
Comment thread diracx-routers/tests/auth/test_standard.py
Comment thread diracx-testing/src/diracx/testing/utils.py
@Robin-Van-de-Merghel Robin-Van-de-Merghel marked this pull request as ready for review June 15, 2025 06:13
Comment thread diracx-logic/src/diracx/logic/pilots/query.py Outdated

@fstagni fstagni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review.

Comment thread diracx-db/src/diracx/db/sql/utils/base.py Outdated
"""Tried to access a value which is asynchronously loaded but not yet available."""


class DiracFormattedError(DiracError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All exceptions apart from the 2 below inherit from DiracError, and this one is further customization for only a few related to pilot. What if instead you modify DiracError?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know really if I should keep as before classes with code duplication, or if a generic thingy will help... I had that in mind, but seems overkill:

class DiracFormattedError(Exception):
     http_status_code = HTTPStatus.BAD_REQUEST  # 400
    pattern = "Error [args]"

    def __init__(self, data: dict[str, str], detail: str = ""):
        self.data = data
        self.detail = detail
        super().__init__(self._render_message())

    def _render_message(self) -> str:
        context = {
            **self.data,
            "args": self._format_args(),
            "detail": self._format_detail(),
        }

        return re.sub(r'\[([^\]]+)\]', lambda m: context.get(m.group(1), ""), self.pattern)

    def _format_args(self) -> str:
        if not self.data:
            return ""
        return " ".join(f"({k}: {v})" for k, v in self.data.items())

    def _format_detail(self) -> str:
        return f": {self.detail}" if self.detail else ""

Then:

class PilotAlreadyExistsError(DiracFormattedError):
    http_status_code = HTTPStatus.CONFLICT 
    pattern = "Pilot [args] already exists[detail]"

Comment thread diracx-db/src/diracx/db/sql/job/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/tests/pilots/test_query.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/tests/pilots/test_pilot_creation.py Outdated
@Robin-Van-de-Merghel

Robin-Van-de-Merghel commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

Where an issue?

Whether we have a base router with require_auth or not, we won't be able to override it in its children (cf #417 ).
So for pilots we must have require_auth=False at the base router because of secret-exchange. But now every pilot endpoint is opened.

Possibilities

Splitting

Let's start with the routers themselves. We can separate pilots and users endpoints : /api/pilots (only for pilots) and ~/api/pilot_management (only for users).

That brings us:

  1. Cleaner: each their own territory, no overlapping, easier to read (you know that on /pilots its only pilot resources)
  2. More secure: we can add a dependency to the pilot router (same as verify_dirac_pilot_token but for pilots)
  3. We can have require_auth for /pilot_management, and enforce tokens

Using another approach

We could have a bare base router without dependency injection, with sub routers with verify_dirac_access_token:

# Authenticated parent router
protected_router = APIRouter(dependencies=[Depends(verify_dirac_access_token)])

@protected_router.get("/secure")
def secure_route():
    return {"msg": "secure"}

# Public sub-router (no auth)
public_router = APIRouter()

@public_router.get("/public")
def public_route():
    return {"msg": "public"}

Its goal would be to replacer DiracxRouter by fixing the issue, and keeping an explicit depedency injection.
Or have @router.authentificated.get(...) instead of @router.get, to explicitely say if we want auth or not

Comment thread diracx-routers/src/diracx/routers/pilots/query.py Outdated
)


async def clear_pilots(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No description provided.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(For deletePilots: delete mapping also, same later for logs #511 + PilotOutput)

@Robin-Van-de-Merghel Robin-Van-de-Merghel Jun 27, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left open as a reminder (already did mapping and output)

Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from a9b353d to 4ba2c9b Compare July 4, 2025 11:48
@aldbr aldbr force-pushed the robin-pilot-management branch 2 times, most recently from cfc7a67 to 411cfe6 Compare April 20, 2026 06:13
@aldbr

aldbr commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

Great, but now there are conflicts, and one test is not passing. Can you fix?

Yes it's fixed now

@aldbr aldbr force-pushed the robin-pilot-management branch from 411cfe6 to 9094f87 Compare April 22, 2026 08:56
@fstagni fstagni linked an issue May 29, 2026 that may be closed by this pull request
@aldbr aldbr force-pushed the robin-pilot-management branch 2 times, most recently from f6d816b to 642003e Compare May 31, 2026 12:55
@aldbr

aldbr commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

pytest-integration not passing, should be fixed with #924 🤞

@aldbr aldbr force-pushed the robin-pilot-management branch from 642003e to c51fef8 Compare June 2, 2026 12:29
@aldbr aldbr requested review from fstagni and removed request for aldbr June 2, 2026 15:28
@fstagni fstagni force-pushed the robin-pilot-management branch from c51fef8 to ce1f640 Compare June 10, 2026 08:10
@aldbr aldbr force-pushed the robin-pilot-management branch from fbc9233 to c0c1ff2 Compare June 10, 2026 09:05
@aldbr

aldbr commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

ddev:
part of a bigger plan with 3 axis: pilot management (user), pilot security (pilot tokens), pilot transition in DIRAC
only missing piece is the getPilotOutput: could be handled with a DIRAC Agent to fetch the pilot output and put it in the DB for now
also the current DIRAC Webapp does not communicate to diracx

There was a plan (may be not written anywhere?):

  • pilot can talk to DIRAC and DiracX
  • DiracX can interact with pilot
    Either we revisit that plan or we write it down in an ADR

@fstagni will double check and see if getPilotOutput is the only issue. If it's the case, we just isolate it in DIRAC (outside of the PilotManagerHandler) and we move on with the legacy adaptor. Will closely be followed by issues if any problem occurs afterwards.



@router.post("/")
async def register_pilots(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to support bulk registration? The SiteDirectors in DIRAC should be accessing the DB directly and only one pilot will appear from the vaccum at a time?



@router.delete("/", status_code=HTTPStatus.NO_CONTENT)
async def delete_pilots(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should only be called via diracx tasts and DIRAC agents which are accessing the DB directly?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but IIRC this is part of the PilotManagerHandler and I guess it's here because of that.
@fstagni can you double check please? Because if it's only used by the webapp / CLI (which I assume is the case), then we can move it to the web utilities for DIRAC as getPilotOutput.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test of pilot database and routers Open access and require auth not working inside a router

4 participants